Automated panning and zooming in teleoperated surgical systems with stereo displays

ABSTRACT

In one embodiment, a digital zoom and panning system for digital video is disclosed including an image acquisition device to capture digital video images; an image buffer to store one or more frames of digital video images as source pixels; a display device having first pixels to display images; a user interface to accept user input including a source rectangle to select source pixels within frames of the digital video images, a destination rectangle to select target pixels within the display device to display images, and a region of interest within the digital video images to display in the destination rectangle; and a digital mapping and filtering device to selectively map and filter source pixels in the region of interest from the image buffer into target pixels of the display device in response to the user input.

CROSS REFERENCE TO RELATED APPLICATIONS

This non-provisional patent application claims the benefit of and is acontinuation application of U.S. patent application Ser. No. 16/034,004,entitled AUTOMATED PANNING IN ROBOTIC SURGICAL SYSTEMS BASED ON TOOLTRACKING filed by Brian Hoffman et al. on Jul. 12, 2018, now allowed.

U.S. patent application Ser. No. 16/034,004 claims the benefit of and isa continuation application of U.S. patent application Ser. No.15/615,830, entitled APPARATUS FOR AUTOMATED PANNING AND ZOOMING INROBOTIC SURGICAL SYSTEMS filed by Brian Hoffman et al. on Jun. 6, 2017,issued as U.S. Pat. No. 10,038,888 on Jul. 31, 2018.

U.S. patent application Ser. No. 15/615,830 claims the benefit of and isa continuation application of U.S. patent application Ser. No.13/408,236, entitled APPARATUS FOR AUTOMATED PANNING AND DIGITAL ZOOMINGIN ROBOTIC SURGICAL SYSTEMS filed by Brian Hoffman et al. on Feb. 29,2012, now issued as U.S. Pat. No. 9,699,445 on Jul. 4, 2017.

U.S. patent application Ser. No. 13/408,236 claims the benefit and is adivisional of U.S. patent application Ser. No. 12/058,663, entitledAUTOMATED PANNING AND DIGITAL ZOOMING FOR ROBOTIC SURGICAL SYSTEMS filedby Brian Hoffman et al. on Mar. 28, 2008, now patented as U.S. Pat. No.8,155,479 on Apr. 10, 2012.

FIELD

The embodiments of the invention relate generally to vision subsystemsfor minimally invasive robotic surgical systems.

BACKGROUND

Minimally invasive surgical (MIS) procedures have become more commonusing robotic (e.g., telerobotic) surgical systems. An endoscopic camerais typically used to provide images to a surgeon of the surgical cavityso that the surgeon can manipulate robotic surgical tools therein.

A surgeon's focus is typically on the tissue or organs of interest in asurgical cavity. He may manually move the endoscopic camera in andaround a surgical site or cavity to properly see and manipulate tissuewith robotic surgical tools. However, when the endoscopic camera ismanually moved inward so that tissue is at desired magnifications,typically a narrow field of view is provided of the surgical cavity bythe endoscopic camera. Tools or tissue that are outside the field ofview typically require the surgeon to manually cause the endoscopiccamera to move to a different position or manually move the camera backout.

Some times the endoscopic camera is slightly moved left, right, up,and/or down to see a slightly different view or slightly moved out toobtain a slightly larger field of view and then moved right back to theoriginal position to the desired magnification to manipulate tissue.

Some times a surgeon may have to initially guess which direction to movethe endoscopic camera to position the tissue and/or tool of interest inthe surgical cavity within the field view of the endoscopic camera.

A more efficient use of the endoscopic camera may also make surgicalprocedures with a robotic surgical system more efficient.

BRIEF SUMMARY

The embodiments of the invention are summarized by the claims thatfollow below.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1A is a block diagram of a robotic medical system including astereo viewer and an image guided surgery (IGS) system with a tooltracking sub-system.

FIG. 1B is a block diagram of a patient side cart including roboticsurgical arms to support and move robotic instruments.

FIG. 1C is perspective view of an endoscopic camera manipulator orrobotic surgical arm.

FIG. 2 is a functional block diagram of the video portion of the IGSsystem to provide a stereo image in both left and right video channelsto provide three-dimensional images in a stereo viewer.

FIG. 3 is a perspective view of a robotic surgical master controlconsole including a stereo viewer and an IGS system with tool trackingsub-system.

FIG. 4A is a cutaway side view of the stereo viewer with gaze detectionin the robotic surgical master control console.

FIG. 4B is a perspective view of the stereo viewer with gaze detectionin the robotic surgical master control console.

FIG. 4C is a side view of the stereo viewer with gaze detection in therobotic surgical master control console.

FIG. 5A is perspective view of a video frame including video images of asurgical site with a navigation window.

FIG. 5B is a schematic view of the video frame including video images ofa surgical site with a navigation window.

FIG. 6A is a perspective view of a video frame including video images ofa surgical site with a digital zoomed fovea portion.

FIG. 6B is an exemplary illustration of a linear mapping between sourcepixel information and target pixels for a digitally zoomed fovea of adisplay and a non-linear mapping between source pixel information andtarget pixels for a background or surround image portion of the display.

FIG. 6C is a schematic diagram illustrating of a linear mapping betweensource pixel information and target pixels for a digitally zoomed foveaof a display and a linear mapping between source pixel information andtarget pixels for a background or surround image portion of the display.

FIG. 6D is a schematic diagram illustrating a mapping between sourcepixel information and target pixels of a display.

FIG. 6E is a schematic diagram illustrating the inner and outer sourcepixel windows of FIG. 6D.

FIG. 6F is an exemplary illustration of a linear mapping between sourcepixel information and target pixels for a digitally zoomed fovea of adisplay and a linear mapping between source pixel information and targetpixels for a background or surround image portion of the display.

FIGS. 7A-7D are diagrams to illustrate combinations of digital panand/or mechanical panning of the endoscopic camera of a frame of a videoinformation with a digital zoom portion in response to gaze detection.

FIG. 8 illustrates a gradual movement of the digital zoom portion overmultiple frames of video information.

FIG. 9 illustrates a face with stereo gaze detection to detect left andright pupil positions.

FIG. 10 illustrates left and rights graphs as to how the position of thepupil may be sensed with respect to the edges of the eye.

FIGS. 11A-11B illustrates a face with an upper left gaze position and alower right left gaze position, respectively.

FIG. 12 illustrates how vertical head movement may be detected.

FIG. 13 illustrates how a combination of vertical and horizontal headmovement may be detected.

FIG. 14 illustrates a touch screen user interface in a display device toprovide a control input to control a robotic surgical instrument such asan endoscopic camera.

FIG. 15 illustrates manual movement of a display device to provide acontrol input to control a robotic surgical instrument such as anendoscopic camera.

FIG. 16 is a functional block diagram of a digital video zoom subsystemto provide digital zoom portion and automatic panning of videoinformation in a surgical site.

FIGS. 17A-17B illustrate a perspective view of an image and automaticpanning of a fovea within the image using a tool centroid.

FIGS. 18A-18B illustrate a perspective view of an image and panning afovea within the image using a robotic surgical tool to poke the foveaaround therein.

DETAILED DESCRIPTION

In the following detailed description of the embodiments of theinvention, numerous specific details are set forth in order to provide athorough understanding of the present invention. However, it will beobvious to one skilled in the art that the embodiments of the inventionmay be practiced without these specific details. In other instances wellknown methods, procedures, components, and circuits have not beendescribed in detail so as not to unnecessarily obscure aspects of theembodiments of the invention.

INTRODUCTION

Aspects of the invention include methods, apparatus and systems forautomated panning and digital zooming for video subsystems of roboticsurgical systems.

High definition endoscopic cameras may generate a greater number ofpixels than can be displayed by liquid crystal display panels or displaymonitors. Aspects of some of the disclosed embodiments of the inventionmay use some of the extra pixel information captured by high definitionendoscopic cameras that would otherwise be unused and possiblydiscarded.

Automatic camera following, an aspect of some embodiments of theinvention, is disclosed that may be responsive to robotic surgicalinstrument location using API information, or selection of an activearea in a surgical site into which the surgeon desires to gaze.

A linear digital zoom, another aspect of some embodiments of theinvention, is disclosed that linearly scales a spatial subset of asource of high definition video images on one or more displays. The fullspatial high definition video images may be linearly scaled down ordown-sampled and displayed picture-in-picture (PIP) as a navigationwindow or a pull-back view for example.

On the same display device, a linear digital zoom of a spatial subset ofthe source the high definition video images may combined with anon-linear digital zoom of another spatial subset of the source of thehigh definition video images, in some embodiments of the invention. Afirst spatial subset of the source of the high definition video imagesmay be digitally zoomed linearly and displayed or rendered in a targetwindow portion (fovea) on a display device and concurrently a secondspatial subset of the source of the high definition video images aroundthe first spatial subset may be digitally zoomed non-linearly anddisplayed or rendered in a target frame portion (background or surround)around the target window portion (fovea) on the display device toprovide a smooth image transition.

The frame portion (background or surround) with the second spatialsubset of the source of the high definition video images altered by anon-linear digital zoom factor may be used to complete the surgeon'sfield of view around the window portion (fovea). In one configuration ofthe invention, the target window portion (fovea) may be displayed inhigh-resolution while the frame portion (background or surround) isdisplayed with a lower-resolution to provide an improved sense ofperipheral vision. With an improved sense of peripheral vision, the needfor a PIP navigation window of the surgical site displayed on thedisplay monitor is reduced. The frame portion (background or surround)with the non-linear digital zoom may reduce the number of otherwisefrequent short duration camera control events. Short duration cameracontrol events are adjustments in the endoscopic camera that are oftenmade due to a surgeon's desire to see what isjust-outside-the-field-of-view or in reaction to lack of peripheralvision, rather than adjustments made to obtain a better field of view ofthe operative site.

Automatic camera following may be combined together with a digital zoomin some embodiments of the invention such that the digital zoomedportion of an image tracks or follow a surgeon's motions, such as thegaze of his pupils, without requiring mechanical movement of theendoscopic camera. If the surgeon's motions indicate that the digitalzoomed portion extend beyond pixels of the high definition digital imagebeing captured, the endoscopic camera may be mechanically moved orpanned automatically.

For automatic camera following, different sensing modalities may be usedto detect a surgeon's motion so that a digital zoomed portion ofinterest of an image may be moved around within the pixels of a highdefinition digital image. Some different sensing modalities include (I)robotic surgical tool tracking, (2) surgeon gaze tracking; (3) or adiscrete user interface. Robotic surgical tool tracking may be performedby kinematics sensing through joint encoders, potentiometers, and thelike; video analysis-based tool location tracking; or a combination orfusion of kinematics sensing and video analysis-based tool locationtracking. A discrete user interface may include one or more of buttonactuation (such as arrow buttons to the side of a surgeon's console),button presses of master console handle buttons, foot-pedal presses, orvoice recognition activation. The discrete user interface may be used tore-center the digital zoomed image based on current tool position, gazelocation, or the like. Alternatively, the discrete user interface may beused to re-center or move the image at discrete times, such as throughvoice activation, perhaps in concert with tool tracking or gazedetection.

Robotic Medical System

Referring now to FIG. 1A, a block diagram of a robotic surgery system100 is illustrated to perform minimally invasive robotic surgicalprocedures on a patient P on an operating table T using one or morerobotic arms 158A-158C (collectively referred to as robotic arms 158).The one or more robotic arms often support a robotic instrument 101. Forinstance, a robotic surgical arm (e.g., the center robotic surgical arm158B) may be used to support a stereo or three-dimensional surgicalimage capture device (endoscopic camera) 101B such as a stereo endoscope(which may be any of a variety of structures such as a stereolaparoscope, arthroscope, hysteroscope, or the like), or, optionally,some other imaging modality (such as ultrasound, fluoroscopy, magneticresonance imaging, or the like).

Robotic surgery may be used to perform a wide variety of surgicalprocedures, including but not limited to open surgery, neurosurgicalprocedures (e.g., stereotaxy), endoscopic procedures (e.g., laparoscopy,arthroscopy, thoracoscopy), and the like.

A user or operator O (generally a surgeon) performs a minimally invasivesurgical procedure on patient P by manipulating control input devices(touch sensitive master control handles) 160 at a master control console150. A computer 151 of the console 150 directs movement of roboticallycontrolled endoscopic surgical instruments (robotic surgical tools orrobotic instruments) 101A-101C via control lines 159, effecting movementof the instruments using a robotic patient-side system 152 (alsoreferred to as a patient-side cart). In a stereo display device 164 ofthe master control console 150, the operator O views video images of thesurgical site including the robotic surgical tools that are in the fieldof view of the endoscopic camera 101B.

The robotic patient-side system 152 includes one or more robotic arms158. Typically, the robotic patient-side system 152 includes at leastthree robotic surgical arms 158A-158C (generally referred to as roboticsurgical arms 158) supported by corresponding positioning set-up arms156. The central robotic surgical arm 158B may support an endoscopiccamera 101B. The robotic surgical arms 158A and 158C to the left andright of center may support robotic instruments 101A and 101C,respectively, that may manipulate tissue.

Robotic instruments (robotic surgical tools) are generally referred toherein by the reference number 101. Robotic instruments 101 may be anyinstrument or tool that couples to a robotic arm that can be manipulatedthereby and can report back kinematics information to the roboticsystem. Robotic instruments include, but are not limited to, surgicaltools, medical tools, bio-medical tools, and diagnostic instruments(ultrasound, computer tomography (CT) scanner, magnetic resonance imager(MRI)).

Generally, the robotic patient-side system 152 includes a positioningportion and a driven portion. The positioning portion of the roboticpatient-side system 152 remains in a fixed configuration during surgerywhile manipulating tissue. The driven portion of the roboticpatient-side system 152 is actively articulated under the direction ofthe operator O generating control signals at the surgeon's console 150during surgery. The driven portion of the robotic patient-side system152 may include, but is not limited or restricted to robotic surgicalarms 158A-158C.

The instruments 101, the robotic surgical arms 158A-158C, and the set upjoints 156,157 may include one or more displacement transducers,positional sensors, and/or orientational sensors 185,186 to assist inacquisition and tracking of robotic instruments. From instrument tip toground (or world coordinate) of the robotic system, the kinematicsinformation generated by the transducers and the sensors in the roboticpatient-side system 152 may be reported back to a tracking system 352 ofthe robotic surgical system.

As an exemplary embodiment, the positioning portion of the roboticpatient-side system 152 that is in a fixed configuration during surgerymay include, but is not limited or restricted to set-up arms 156. Eachset-up arm 156 may include a plurality of links and a plurality ofjoints. Each set-up arm may mount via a first set-up-joint 157 to thepatient side system 152.

An assistant A may assist in pre-positioning of the robotic patient-sidesystem 152 relative to patient P as well as swapping tools orinstruments 101 for alternative tool structures, and the like, whileviewing the internal surgical site via an external display 154. Theexternal display 154 or some other external display may be positioned orlocated elsewhere so that images of the surgical site may be displayedto students or other interested persons during a surgery. Images withadditional information may be overlaid onto the images of the surgicalsite by the robotic surgical system for display on the external display154.

Referring now to FIG. 1B, a perspective view of the robotic patient-sidesystem 152 is illustrated. The robotic patient-side system 152 comprisesa cart column 170 supported by a base 172. One or more robotic surgicalarms 158 are respectively attached to one or more set-up arms 156 thatare a part of the positioning portion of robotic patient-side system152. Situated approximately at a central location on base 172, the cartcolumn 170 includes a protective cover 180 that protects components of acounterbalance subsystem and a braking subsystem (described below) fromcontaminants.

Excluding a monitor arm 158E for the monitor 154, each robotic surgicalarm 158 is used to control robotic instruments 101A-101C. Moreover, eachrobotic surgical arm 158 is coupled to a set-up arm 156 that is in turncoupled to a carriage housing 190 in one embodiment of the invention, asdescribed below with reference to FIG. 3. The one or more roboticsurgical arms 158 are each supported by their respective set-up arm 156,as is illustrated in FIG. 1B.

The robotic surgical arms 158A-158D may each include one or moredisplacement transducers, orientational sensors, and/or positionalsensors 185 to generate raw uncorrected kinematics data, kinematicsdatum, and/or kinematics information to assist in acquisition andtracking of robotic instruments. The robotic instruments may alsoinclude a displacement transducer, a positional sensor, and/ororientation sensor 186 in some embodiments of the invention. Moreover,one or more robotic instruments may include a marker 189 to assist inacquisition and tracking of robotic instruments.

Robotic Surgical Arms

Referring now to FIG. 1C, a perspective view of the robotic surgical arm158B is illustrated. As discussed previously, the center roboticsurgical arm 158B is for coupling to an endoscopic camera 101B. Theendoscopic camera 101B may not have an end effector that requirescontrolling. Thus, fewer motors, cables, and pulleys may be employed incontrolling the endoscopic camera 101B. However for the purposes ofoverall movement (e.g., pitch, yaw, and insertion), the elements of thecenter robotic surgical arm 158B are similar to the elements of therobotic surgical arms 158A,158C.

In robotic surgical systems for minimally invasive surgery, it isdesirable to move and constrain a robotic surgical tool at a singlefixed remote center point 556. Typically the fixed remote center point556 is near the point of insertion of the surgical tool into the patientP. The center of rotation 556 may be aligned with the incision point tothe internal surgical site, for example, by a trocar or cannula at anabdominal wall during laparoscopic surgery. As the fixed remote centerpoint 556 is on the insertion axis 574 of the robotic camera and isoffset and remote from ground, the robotic surgical arm may also bereferred as an offset remote center manipulator instead.

The robotic surgical arm 158B includes serial links 541-545 pivotallycoupled in series at joints 512-514 near respective ends of the links.The first link (Link 1) 541 is pivotally coupled to a drive mount 540 ata first joint 511 near a first end and the second link (Link 2) 542 atthe second joint 512 near a second end. The third link (Link 3) 543 ispivotally coupled to the second link 542 near a first end and pivotallycoupled to the fourth link (Link 4) 544 near a second end. Generally,the fourth link 544 is substantially in parallel to the insertion axis574 of the endoscopic camera 101B. A fifth link (Link 5) 545 isslidingly coupled to the fourth link 544. The endoscopic camera 101Bmounts to the fifth link 545 as shown.

The robotic surgical arm 158B further includes a mounting base 540 thatallows it to be mounted and supported by set-up arms/joints of a patientside system. The mounting base 540 is pivotally coupled to the firstlink 541 and includes a first motor 551 to yaw the robotic surgical armabout a yaw axis at the pivot point. The second link 542 houses a secondmotor 552 to drive and pitch the linkage of the arm about a pitch axisat the pivot point 556. The fourth link 544 may include a third motor553 to slide the firth link 545 and the endoscopic camera 101B along theinsertion axis 574.

The robotic endoscopic camera arm 158B and the robotic surgical arms158A,158C have a drive train system driven by the motors 551-553 tocontrol the pivoting of the links about the joints 512-514. If theendoscopic camera 101B is to be mechanically moved, one or more of themotors 551-553 coupled to the drive train are energized to move thelinks of the robotic endoscopic camera arm 158B. Other tools 101attached to the robotic surgical arms 158A,158C may be similarly moved.

Endoscopic Video System

Referring now to FIG. 2, the stereo endoscopic camera 101B includes anendoscope 202 for insertion into a patient, a camera head 204, a leftimage forming device (e.g., a charge coupled device (CCD)) 206L, a rightimage forming device 206R, a left camera control unit (CCU) 208L, and aright camera control unit (CCU) 208R coupled together as shown. Thestereo endoscopic camera 101B generates a left video channel 220L and aright video channel 220R of frames of images of the surgical sitecoupled to a stereo display device 164 through a video board 218. Toinitially synchronize left and right frames of data, a lock referencesignal is coupled between the left and right camera control units208L,208R. The right camera control unit generates the lock signal thatis coupled to the left camera control unit to synchronize the left viewchannel to the right video channel. However, the left camera controlunit 208L may also generate the lock reference signal so that the rightvideo channel synchronizes to the left video channel.

The stereo display device 164 includes a left monitor 230L and a rightmonitor 230R. As discussed further herein, the viewfinders or monitors230L,230R may be provided by a left display device 402L and a rightdisplay device 402R, respectively. The stereo images may be provided incolor by a pair of color display devices 402L,402R.

Additional details of a stereo endoscopic camera and a stereo displaymay be found in U.S. Pat. No. 5,577,991 entitled “Three DimensionalVision Endoscope with Position Adjustment Means for Imaging Device andVisual Field Mask” filed on Jul. 7, 1995 by Akui et al; U.S. Pat. No.6,139,490 entitled “Stereoscopic Endoscope with Virtual Reality Viewing”filed on Nov. 10, 1997 by Breidenthal et al; and U.S. Pat. No. 6,720,988entitled “Stereo Imaging System and Method for use in TeleroboticSystems” filed on Aug. 20, 1999 by Gere et al.; all of which areincorporated herein by reference. Stereo images of a surgical site maybe captured by other types of endoscopic devices and cameras withdifferent structures. For example, a single optical channel may be usedwith a pair of spatially offset sensors to capture stereo images of thesurgical site.

Referring now to FIG. 3, a perspective view of the robotic surgicalmaster control console 150 is illustrated. The master control console150 of the robotic surgical system 100 may include a computer 151, astereo viewer 312, an arm support 314, a pair of control input wristsand control input arms in a workspace 316, foot pedals 318 (includingfoot pedals 318A-318B), and a head sensor 320. The master controlconsole 150 may further include a digital zoom/panning system 351 and atracking system 352 coupled to the computer 151 for providing thedigital zoomed images, fovea images, and/or PIP images of the surgicalsite. The tracking system 352 may be a tool tracking system or a surgeonmotion tracking system, such as for gaze detection/tracking, to providefor the digital panning of the camera images.

The stereo viewer 312 has two displays where stereo three-dimensionalimages of the surgical site may be viewed to perform minimally invasivesurgery. When using the master control console, the operator O typicallysits in a chair, moves his or her head into alignment with the stereoviewer 312 to view the three-dimensional images of the surgical site. Toensure that the operator is viewing the surgical site when controllingthe robotic instruments 101, the master control console 150 may includea head sensor 320 disposed adjacent the stereo viewer 312. When thesystem operator aligns his or her eyes with the binocular eye pieces ofthe stereo viewer 312 to view a stereoscopic image of the surgicalworksite, the operator's head activates the head sensor 320 to enablethe control of the robotic instruments 101. When the operator's head isremoved from the area of the stereo viewer 312, the head sensor 320 isdeactivated to disable or stop generating new control signals inresponse to movements of the touch sensitive master control handles 160in order to hold the state of the robotic instruments.

The arm support 314 can be used to rest the elbows or forearms of theoperator O (typically a surgeon) while gripping touch sensitive mastercontrol handles 160 of the control input wrists, one in each hand, inthe workspace 316 to generate control signals. The touch sensitivemaster control handles 160 are positioned in the workspace 316 disposedbeyond the arm support 314 and below the viewer 312. This allows thetouch sensitive master control handles 160 to be moved easily in thecontrol space 316 in both position and orientation to generate controlsignals. Additionally, the operator O can use his feet to control thefoot-pedals 318 to change the configuration of the surgical system andgenerate additional control signals to control the robotic instruments101 as well as the endoscopic camera.

The computer 151 may include one or more microprocessors 302 to executeinstructions and a storage device 304 to store software with executableinstructions that may be used to generate control signals to control therobotic surgical system 100. The computer 151 with its microprocessors302 interprets movements and actuation of the touch sensitive mastercontrol handles 160 (and other inputs from the operator O or otherpersonnel) to generate control signals to control the robotic surgicalinstruments 101 in the surgical worksite. In one embodiment of theinvention, the computer 151 and the stereo viewer 312 map the surgicalworksite into the controller workspace 316 so it feels and appears tothe operator that the touch sensitive master control handles 160 areworking over the surgical worksite. The computer 151 may couple to thedigital zoom/panning system 351 and the tracking system 352 to executesoftware and perform computations for the digital zoom/panning system.

Referring now to FIG. 4A, a side cutaway view of the surgeon's mastercontrol console 150 is shown to illustrate the stereo viewer 312 with agaze detection/tracking system. The stereo viewer 312 may include a leftdisplay 402L and one or more left gaze detection sensors 420L for theleft eye EL of a surgeon and a right display 402R and one or more rightgaze detection sensors 420R (not shown in FIG. 4A, see FIG. 4B) for theright eye of the surgeon. The head sensor 320 illustrated in FIG. 3 maybe used to enable/disable the gaze detection system so that other motionis not inadvertently sensed as the surgeon's eye movement.

FIG. 4C illustrates a magnified side view of the stereo viewer 312including the left display 402L and the one or more left gaze detectionsensors 420L for the left eye EL of the surgeon. The one or more leftgaze detection sensors 420L may sense X and Y axes movement of a pupilPL along a Z optical axis.

A fixed lens 450 may be provided between each eye and each respectivedisplay device 402L,402R to magnify or adjust the apparent depth of thedisplayed images I over a depth range 452. The focus on an image in thesurgical site is adjusted prior to image capture by a moveable lens inthe endoscopic camera 101B that is in front of the CCD image sensor.

Referring now to FIG. 4B, a perspective view of the stereo viewer 312 ofthe master control console 150 is illustrated. To provide athree-dimensional perspective, the viewer 312 includes stereo images foreach eye including a left image 400L and a right image 400R of thesurgical site including any robotic instruments 101 respectively in aleft viewfinder 401L and a right viewfinder 401R. The images 400L and400R in the viewfinders may be provided by a left display device 402Land a right display device 402R, respectively. The display devices402L,402R may optionally be pairs of cathode ray tube (CRT) monitors,liquid crystal displays (LCDs), or other type of image display devices(e.g., plasma, digital light projection, etc.). In the preferredembodiment of the invention, the images are provided in color by a pairof color display devices 402L,402R, such as color CRTs or color LCDs.

In the stereo viewer 312, three dimensional images of a navigationwindow or a fovea may be rendered within the main image of the surgicalsite. For example, in the right viewfinder 401R a right navigationwindow image 410R may be merged into or overlaid on the right image 400Rbeing displayed by the display device 402R. In the left viewfinder 401L,a left navigation window image 410L may be merged into or overlaid onthe left image 400L of the surgical site provided by the display device402L.

If the gaze detection system is used to control the position of thefovea or the digital panning of the digital zoom image of the surgicalsite, the stereo viewer 312 may include one or more left gaze detectionsensors 420L near the periphery of the display device 402L for the lefteye of the surgeon and one or more right gaze detection sensors 420Rnear the periphery of the display device 402R for the right eye of thesurgeon. One of the gaze detection sensors for each eye may also includea low level light source 422L,422R to shine light into the eye of thesurgeon to detect eye movement with the respective gaze detectionsensors 420L,420R.

While a stereo video endoscopic camera 101B has been shown anddescribed, a mono video endoscopic camera generating a single videochannel of frames of images of the surgical site may also be used in anumber of embodiments of the invention. Images, such as a navigationwindow image, can also be overlaid onto a portion of the frames ofimages of the single video channel.

Digital Zoom

As the endoscopic camera 101B is a digital video camera, it providesdigital pixel information regarding the images that are captured. Thus,the digital images that are captured may be digitally zoomed in order tobring the objects closer in into view in the display of an image. In analternate embodiment of the invention, the endoscopic camera 101B mayinclude an optical zoom, in addition to a digital zoom, to magnifyobjects prior to image capture by using mechanical movement of optics,such as lenses.

In contrast to an optical zoom that involves a movement of optics, adigital zoom is accomplished electronically without any adjustment ofthe optics in the endoscopic camera 101B. Generally, a digital zoomselects a portion of an image and manipulates the digital pixelinformation, such as interpolating the pixels to magnify or enlarge theselected portion of the image. In other words, a digital zoom may crop aportion of an image and then enlarge it by interpolating the pixels toexceed the originally cropped size. While the cropped image may belarger, a digital zoom may decrease or narrow an apparent angle of viewof the overall video image. To the surgeon, a digitally zoomed imagealone may have a reduced field of view of the surgical site. Otherimages may be provided to compensate for the reduced field of view inthe digitally zoomed image.

With some embodiments of invention, a region-of-interest is selectedfrom source video images to undergo a digital zoom. The selected regionof interest is then scaled linearly for presentation to the display(e.g., as a fovea 650). The region of interest may be scaled up(interpolated), or scaled down (decimated), depending on the number ofpixels in the source region-of-interest, relative to the number ofpixels allocated (for this tile of video) on the display. Digitalfiltering of the source data is performed as part of theinterpolation/decimation process. Selection of a region-of-interestsmaller than the full source video frame reduces the surgeon's effectivefield of view into a surgical site.

Note that there are four degrees of freedom available to a digitalzoomed image in a rigid endoscope. The embodiments of the invention maypan a digital zoomed image up, down, left, and/or right and it mayrotate the image and/or change its level of zoom.

As discussed previously herein, the endoscopic camera 101B is a highdefinition camera. In one embodiment of the invention, the highdefinition endoscopic camera 101B has a greater resolution than theresolution of the display devices 402L,402R. The extra pixel informationfrom the high definition endoscopic camera 101B may be advantageouslyused for digital zoom. The region of interest selected from the sourcevideo need not be mapped one-to-one or magnified. In fact, a region ofinterest selected from the source video may contain more pixels than areallocated on the display for presentation of the video source. If thatis the case, the pixels in the selected region of interest may be scaleddown (decimated), while still appearing to the user to zoom in onobjects.

Texture mapping, pixel mapping, mapping pixels, or mapping texturepixels, may be used interchangeably herein as functional equivalentswhere a source image is sampled at source coordinates or points(t_x,t_y) and a target image is colored at target coordinates or points(v_x,v_y).

As discussed previously, one aspect of some embodiments of the inventionmay be a linear digital zoom while one aspect of some embodiments of theinvention may be a non-linear digital zoom.

Referring now to FIG. 5A, a perspective view of images 500 in the stereoviewer 312 with a linear digital zoom is illustrated. A linear digitalzoomed view 501 is displayed in a substantial portion of the display402L,402R. The linear digital zoomed view 501 may magnify the images oftissue 505 and a right side surgical tool 510R in the surgical site.Alternatively, the view 501 may be a spatial subset of high definitionimages displayed on a portion of the display 402L,402R.

Within the linear digital zoomed view 501 may be a navigation window orpull-back view 502. The navigation window or pull-back view 502 may bethe full spatial high definition image that has been down-sampled to bedisplayed picture-in-picture (PIP) within the smaller display region.

Referring now to FIG. 5B, a pixel map diagram is illustrated for thelinear digital zoomed view 501 of FIG. 5A. The stereo endoscopic camera101B captures left and right high definition spatial images 510 with atwo dimensional array of pixels that is HDX pixels wide by HDY pixelshigh. For example, the two dimensional array of pixels for the highdefinition spatial images 510 may be 1920 pixels wide by 1080 pixelshigh.

However, the display devices 402L,402R in the stereo view 312 may onlydisplay low definition images 511N with a two-dimensional array ofpixels with a native resolution of LDX pixels wide by LDY pixels highthat are respectively less than the available spatial resolution of HDXpixels wide by HDY pixels high for the high definition spatial images510. For example, the two dimensional array of pixels for the lowdefinition spatial images 511N may be 1280 pixels wide (LDX) by 1024pixels high (LDY) in contrast to 1920 pixels wide (HDX) by 1080 pixelshigh (HDY) for exemplary high definition spatial images 510.

As the display devices 402L,402R in the stereo viewer 312 display alower native resolution of LDX pixels wide by LDY pixels high, some ofthe pixel information in the full spatial high definition image 510 maygo unused. For example, the position and relationship between the lowdefinition images 511N and the high definition images 510 may be fixed.In which case, pixels 521 within the resolution of the low definitionimage 511N may be displayed on the display devices 402L,402R while somepixels 520 outside the resolution of the low definition image 511N maynot be displayed. In this case, the display devices may be considered asproviding a field of view of a virtual camera inside the endoscopiccamera.

The field of view of the virtual camera within the field of view of theendoscopic camera may be digitally adjusted. That is, the pixels in thehigh definition images 510 that are to be displayed by the displaydevices 402L,402R may be user selectable. This is analogous to the lowdefinition image 511N being a window that can be moved over the array ofHDX by HDY pixels of the high definition spatial image 510 to select anarray of LDX by LDY pixels to display. The window of the low definitionimage 511N may be moved in X and Y directions to select pixels in thearray of HDX by HDY pixels of the high definition spatial image 510. Thepixels in the high definition images 510 that are to be displayed by thedisplay devices 402L,402R may also be digitally manipulated.

A smaller subset of pixels (SX by SY) in the array of HDX by HDY pixelsof the high definition spatial image 510 may be respectively selected bya user for magnification into a digital zoom image 511M. The array of SYpixels high by SX pixels wide of the digital zoom image 511M may beinterpolated with a digital filter or sampling algorithm into a largernumber of pixels of the array of LDX by LDY pixels to display amagnified image on the display devices 402L,402R. For example, 840pixels wide by 672 pixels high may be magnified and expanded to 1280pixels wide by 1024 pixels high maintaining the same aspect ratio fordisplay, such as on the display devices 402L,402R.

While the digital zoom image 511M may be expanded by interpolation intoa larger number of pixels to display a magnified image, such as image501 illustrated in FIG. 5A, the image resolution of the array of HDX byHDY pixels of the high definition spatial image 510 may decimated orreduced down (down-sampled) to shrink or demagnify its image to fit intoa window array 512 of reduced pixels RX pixels high by RY pixels wide tobe used for the navigation window 502 illustrated in FIG. 5A. Forexample, high definition spatial images 510 with an array of 1920 pixelswide by 1080 pixels high may be decimated by a factor of ten to ademagnified image array of 192 pixels wide by 108 pixels high.

While the digital zoom for a portion of the display may have a linearrelationship with the pixels of the full spatial image, the digital zoommay also have a non-linear relationship with the pixels of the fullspatial image in another portion of the display device.

Referring now to FIG. 6A, a perspective view of an image 600 in thestereo viewer 312 with is illustrated. A digital zoomed portion (fovea)650 is displayed within a background or surround portion 651 of theimage 600 on the display devices 402L,402R. As the digital zoomed view650 may be the focus of the central vision of a surgeon's eyes andsurrounded by the surround 651, the digital zoomed view 650 may also bereferred to as a fovea 650. The digital zoomed view 650 may beconsidered to be a virtual image within a larger image analogous to thevirtual camera within the endoscopic camera.

In FIG. 6A, the digital zoomed view 650 is moveable around the display(moveable fovea) and may magnify the images of tissue 605 and surgicaltools 610R in the surgical site. In another configuration, the digitalzoomed view or fovea 650 is centrally fixed in position (fixed fovea)within the center of the display device (e.g., see FIG. 6B). While thefovea may provide a digitally zoomed image or view of the surgical site,the background or surround image 651 may provide an improved sense ofperipheral vision to the surgeon, possibly reducing or eliminating theneed for one or more navigation windows.

The fovea 650 is formed by a first mapping of first array or set ofsource pixel information (source pixels) from the high definition sourcevideo images to a first array or set of pixels in the display device(target pixels). The surround 651 around the fovea 650 is formed by asecond mapping of a second array or set of source pixel information(source pixels) from the high definition source video images to a secondarray or set of pixels in the display device (target pixels).

The second mapping differs from the first mapping. In one embodiment ofthe invention, the first mapping is a linear mapping and the secondmapping is a non-linear mapping (e.g., see FIG. 6B). In anotherembodiment of the invention, the first mapping and the second mappingare linear mappings (e.g., see FIG. 6F) but differ in other ways, suchas size and/or resolution. For example, the digital zoomed view 650 maybe a high resolution or high definition image while the background orsurround image 651 is a low resolution or low definition image.

The digital zoomed view 650 and the background or surround portion 651of the image 600 are displayed in real time to a surgeon over acontinuing series of video frame images on the displays 402L,402R of thestereo viewer. The images may be continuously updated to view currenttool positions and current state of the surgical site and any tissuethat is being manipulated therein.

At its edges, there may be a sharp or gradual transition from thedigital zoomed view 650 to the background or surrounding image 651. Forease of discussion herein, a sharp or hard edge between the fovea 650and the background 651 may be assumed.

The digital zoomed view 650 may be provided by a linear digital zoomfactor over the given field of view selected by a surgeon to reducedistortion of the image displayed in the fovea 650. The surround view orimage 651 may be provided by a linear digital zoom factor (linearmapping) or a non-linear digital zoom factor (non-linear mapping) overthe given field of view selected.

The size of the digital zoom view 650 within the image 600 may be userselectable by a surgeon at the master control console 150 or by anassistant at the external display 154. That is, a user may selectivelyexpand or contract the x-axis FX and the y-axis FY pixel dimensions ofthe area of the fovea or linear digital zoom view 650. The digital zoomview 650 may be centered in the display to be in line with a centralgaze of the surgeon's eyes. Alternatively, a user may selectivelyposition the linear digital zoom view 650 within different positions onthe display within the image 600 by different user interface meansdescribed herein.

Additionally, the source region-of-interest (source zoom pixels)selected for the fovea 650 from the high definition source video imagesand the source region-of-interest (source background pixels) selectedfrom the high definition source video images for the surround 651 may beadjusted by the user. For example, the source pixels for the backgroundaround the fovea 650 may selected to be a spatial subset of the highdefinition source images. Alternatively, the source pixels for thebackground 651 may be selected to be a set of source pixels to completethe full spatial image of the high definition images. With a largerfield of view provided by the background 651 around the fovea 650, asurgeon's peripheral vision of the surgical site may be improved. Thiscan help avoid or reduce frequent short duration camera control eventsthat otherwise may be made due to a desire to see what's just outsidethe field of view.

As discussed previously, the fovea 650 is formed by a first mapping ofarray or set of source pixel information (source pixels) from the highdefinition source video images to a first array or set of pixels in thedisplay device (target pixels) and the surround 651 is formed by asecond mapping of a second array or set of source pixel information(source pixels) from the high definition source video images to a secondarray or set of pixels in the display device (target pixels).

Referring now to FIG. 6D, mapping functions for the first and secondpixel mappings are determined between coordinates in the source(texture) 660 and coordinates on the target 670 (e.g., display402L,402R,154). Pixel data is mapped from an inner/outer pair of sourcewindows 661 to an inner/outer pair of target windows 671.

The source coordinate system origin 665 is defined to be the upper leftcorner of the source frame 660 with positive-x right, and positive-ydown. The inner source window 663 may be defined by selection of aleft-top coordinate (t_iL,t_iT) 667 and a right-bottom coordinate(t_iR,t_iB) 668. The outer source window 664 may be defined by itsleft-top coordinate (t_oL,t_oT) 666 and right-bottom coordinate(t_oR,t_oB) 669. In the parenthetical coordinate description, the prefixt denotes texture, i/o refers to inner/outer, and L,T,R,B refers toleft, top, right, and bottom, respectively. The coordinates for theinner source window 663 and the outer source window 664 may be directlyor indirectly and automatically or manually selected by a user (e.g.,surgeon O or assistant A) in a number of ways.

The target coordinate system origin 675 is defined to be the upper leftcorner of the target frame 670, with positive-x right and positive-ydown. The inner target window 673 is defined by its left-top coordinate(v_iL,v_iT) 677 and its right bottom coordinate (v_iR,v_iB) 678. Theouter target window 674 is defined by its left-top coordinate(v_oL,v_oT) 676 and its right-bottom coordinate (v_oR,v_oB) 679. In theparenthetical coordinate description, the prefix v denotes vertex, i/orefers to inner/outer, and L,T,R,B refers to left, top, right, andbottom, respectively. The coordinates for the inner target window 673and the outer target window 674 may also be directly or indirectly andautomatically or manually selected by a user (e.g., surgeon O orassistant A) in a number of ways.

Referring now to FIGS. 6D-6E, the region corresponding to the fovea 650is simply formed by linearly scaling the source pixel array 680 of theinner source window 663 from coordinate (t_iL,t_iT) 667 throughcoordinate (t_iR,t_iB) 668 into the target pixel array (fovea) 650 ofthe inner target window 673 from coordinate (v_iL,v_iT) 677 throughcoordinate (v_iR,v_iB) 678. Constructing the surround region 651 aroundthe fovea 650 remains.

The task of mapping source pixels in the frame shaped region 681 betweenthe inner source window 663 and the outer source window 664 into targetpixels in the frame shaped surround region 651 between the inner targetwindow 673 and the outer target window 674 is more difficult due to theframe like shape of each.

Referring now to FIG. 6E, the source pixels in the frame shaped region681 between the inner source window 663 and outer source window 664 issubdivided into a number of N rectangular regions (quads). The Nrectangular regions may be eight (8) rectangular regions, for example.Starting at the upper left hand corner and working clockwise, the eightrectangular regions may be formed by coordinates 666,686,667,688;686,687,683,667; 687,685,692,683; 683,692,693,668; 668,693,669,691;682,668,691,690; 689,682,690,684; and 688,667,682,689. Values for t_x1,t_x2, t_y1, and t_y2 in the coordinate (t_x1,t_oT) 686, coordinate(t_x2,t_oT) 687, coordinate (t_oL,t_y1) 688, coordinate (t_oL,t_y2) 689,coordinate (t_x1,t_oB) 690, coordinate (t_x2,t_oB) 691, coordinate(t_oR,t_y1) 692, and coordinate (t_oR,t_y2) 693 are determined whichallow the subdivision of the frame shaped surround region 681 into the 8rectangular regions (quads).

Referring now to FIGS. 6D-6E, if the source pixels t_oL through t_oR ontop and bottom edges of outer source window 664 are mapped linearly intothe target pixels v_oL through v_oR on top and bottom edges of outertarget window 674, then the values of t_x1 and t_x2 are respectivelyproportional to the length of the line segments from pixels v_oL throughv_iL and pixels v_oL through v_iR along top and bottom edges of theouter source window 664, and may be computed by equations 1 and 2 asfollows:

t_x1=t_oL+(t_oR−t_oL)*((v_iL−v_oL)/(v_oR−v_oL))  (1)

t_x2=t_oL+(t_oR−t_oL)*((v_iR−v_oL)/(v_oR−v_oL))  (2)

Similarly, if the source pixels t_oT through t_oB on the right and leftedges of outer source window 664 are mapped linearly into the targetpixels v_oT through v_oB on left and right edges of outer target window674, then the values of t_y1 and t_y2 are respectively proportional tothe length of the segments from pixels v_oT through v_iT, and pixelsv_oT through v_iB along left and right edges of the outer source window664. Thus, the values of t_y1 and t_y2 may be computed by equations 3and 4 as follows:

t_y1=t_oT+(t_oB−t_oT)*((v_iT−v_oT)/(v_oB−v_oT))  (3)

t_y2=t_oT+(t_oB−t_oT)*((v_iB−v_oT)/(v_oB−v_oT))  (4)

Thus, the source pixels along the edges of the quads may be mapped witha predetermined mapping (e.g., equations 1-4) into target pixels values.

For each interior pixel point (v_x,v_y) in the surround 651 of each quadof the N quads in the source frame 681, we may perform an interpolationto map source pixels into respective t_x and t_y values of the targetpixels. The interpolation may be a non-linear interpolation, such as abilinear interpolation (BI), or a linear interpolation, where theselection of the interpolation function is arbitrary. At larger zoomfactors of the fovea 650, a non-linear interpolation may distort lessthan a linear interpolation.

A quad drawn counter-clockwise, has target vertex coordinates definedas:

Lower Left: v_L, v_B

Lower Right: v_R, v_B

Upper Right: v_R, v_T

Upper Left: v_L, v_T

and associated source texture coordinates defined as:

Lower Left: t_LLx, t_LLy

Lower Right: t_LRx, t_LRy

Upper Right: t_URx, t_URy

Upper Left: t_ULx, t_ULy

For each interior target point v_x,v_y within each quad, the associatedsource texture point t_x, t_y is found by interpolation. With the sourcetexture point or coordinate being known for the source pixel, thetexture of the source texture point can be sampled using an arbitraryfilter function and the target pixel at the target coordinate can becolored with the sampled value of texture. That is, the source textureis sampled at coordinate (t_x,t_y) using a filter function to color thetarget pixel (v_x,v_y). The filter function used in the sampling processmay be arbitrarily complicated but consistently used.

Assuming that a bilinear interpolation (BI) is performed for eachinterior pixel point (v_x,v_y) in the surround 651, we may perform abilinear interpolation (BI) into respective t_x and t_y values(generally referred to as t values) which are specified on the quadboundary by equations 5 and 6 as:

t_x=BI[v_x,v_y;v_L,v_T,v_R,v_B;t_LLx,t_LRx,t_URx,t_ULx]  (5)

t_y=BI[v_x,v_y;v_L,v_T,v_R,v_B;t_LLy,t_LRy,t_URy,t_ULy]  (6)

where t_x and t_y are the interpolated t values at each point (v_x,v_y);v_L,v_T, v_R,v_B are target boundary coordinates; andt_LLx,t_LRx,t_URx,t_ULx are the lower-left, lower-right, upper-right,and upper-left ‘t’ coordinates in x and t_LLy,t_LRy,t_URy,t_ULy are thelower-left, lower-right, upper-right, and upper-left ‘t’ coordinates iny. A bilinear interpolation (BI) is an interpolating function of twovariables on a regular grid. With the values of t_x1, t_x2, t_y1, andt_y2 being known from equations 1-4, there are known coordinates 686-692along the edges of the outer source window 664 that may be used as knownpoints for the interpolation within each of the N quads.

The bilinear interpolation BI( ) may be defined in pseudo code as:

BI(v_x,v_y, v_L,v_T,v_R,v_B, t_LL,t_LR,t_UR,t_UL) { a1 = lerp(v_x, v_L,v_R, t_LL, t_LR); a2 = lerp(v_x, v_L, v_R, t_UL, t_UR); b1 = lerp(v_y,v_T, v_B, a2, a1);//NOTE: swap a2,a1 due to Y+ downward return(b1); }

with lerp( ) being defined in pseudo code as:

lerp(v, v1, v2, q1, q2) { return( q1*((v2−v)/(v2−v1)) +q2*((v−v1)/(v2−v1)) ); }

A bilinear interpolation (BI) is a well known non-linear mathematicalfunction. It is non-linear as it is mathematically proportional to aproduct of two linear functions such as (a₁x+a₂) (a₃y+a₄). In this case,the bilinear interpolation is a combination of multiple linearinterpolations over a grid to smoothly transition images between theinner and outer areas of interest of the source windows 661 and targetwindows 671. The bilinear interpolation results in a quadratic warp inthe surround 651 around the fovea 650.

For example in FIG. 6E, consider the upper left quad of source pixels inthe source frame 681 and mapping them into upper left quad of thesurround 651. The source texture coordinates assigned to each of thefour vertices of the quad of source pixels is determined in accordancewith equations 1-4 described herein. For the upper left quad thefollowing mapping of vertices is determined:

(t_oL,t_y1) maps to (v_oL,v_y1)

(t_iL,t_y1) maps to (v_iL,v_y1)

(t_iL,t_oT) maps to (v_iL,v_oT)

(t_oL,t_oT) maps to (v_oL,v_oT)

Then the texture coordinate (t_x,t_y) of each pixel interior to the quadat position (v_x,v_y) is found via bilinear interpolation. The sourcetexture is sampled at coordinate (t_x,t_y) to color the pixel (v_x,v_y)with an arbitrary filter function.

Each of the N quads is similarly processed once the texture coordinateshave been assigned to its vertices. As adjacent quads have the sametexture coordinates assigned to their shared vertices, the final imageappears to be a smooth warp, without discontinuity acrossquad-boundaries.

Referring now to FIG. 6B, the results of a first linear mapping of acheckerboard pattern into the fovea 650 and a non-linear mapping (e.g.,using bilinear interpolation) of a checkerboard pattern into eight quadsof the surround 651 are illustrated. Lines in the checkerboard of thesource image illustrated on the display indicate warped pixelinformation. As the lines are straight and equidistant in the fovea 650,it is digitally zoomed without any mapping distortion being added. Thesurround 651 experiences some warping as it changes from the digitallyzoomed (magnified) image at the edge of the fovea 650 to a lowerdigitally zoomed (magnified) image at the outer edges of the surround.The warpage in the surround 651 is more noticeable at the corners of thefovea in the FIG. 6B as indicated in the bending lines in thecheckerboard.

Instead of a non-linear mapping between source pixels and the targetpixels in the N quads of the source frame 681, a linear mapping may beused but differs from the linear mapping of pixels for the fovea 650.The mapping of the source pixels in the source frame 681 to the targetpixels in the surround 651 is piecewise linear for the N quads if thevalues of t_x1, t_x2, t_y1, and t_y2 are set as follows:

t_x1=t_iL;

t_x2=t_iR;

t_y1=t_iT;

t_y2=t_iB;

That is, each of the pixels in the N quads is linearly mapped with alinear scaling function into pixels in the surround 651.

Referring now to FIG. 6F, the results of a first linear mapping of acheckerboard pattern into the fovea 650 and a second linear mapping(e.g., piecewise linear) of a checkerboard pattern into eight quads ofthe surround 651 are illustrated. At relatively low digital zoom factorsfor the fovea 650, the surround 651 shows only nominal warpage. Howeverif a relatively high digital zoom factor is applied to the fovea 650 tohighly magnify objects in the fovea 650, the surround 651 with no changein digital zoom factor experiences significant warpage. Thus, it hasbeen determined that a non-linear mapping between source pixels of theframe 681 to target pixels in the surround 651 is preferable.

Note that the resolution of the fovea 650 and the surround 651 dependsupon the selection of the relative sizes of the inner/outer sourceregions and the selection of the relative sizes of the inner/outerdisplay or target regions. If a user selects to digitally zoom the fovea650, the size of the inner source window 663 is typically decreased bychanging a digital zoom factor magnifying the image in the fovea 650. Inthis case, the size of the frame 681 of the source video will changeresulting in a change in the warp of the surround 651 as well.

With the first and second mappings determined from source to target forthe fovea 650 and the surround 651, various digital filter methods andresampling algorithms may then be used to sample the source pixeltexture information for interpolation/decimation into the target pixelsof one or more display devices. Exemplary digital filters that may beused are a box filter, tent filter, Gaussian filter, sinc filter, andlanczos filter.

Referring now to FIG. 6C, a schematic diagram illustrates another linearmapping of source pixels from the high definition video source images ofthe endoscopic camera to target pixels of the display are shown tofurther explain a linear mapping of the fovea 650 and a linear mappingof the surround or background 651.

As discussed previously with reference to FIG. 5B, the high definitionspatial images 510 have a two dimensional array of pixels that is HDXpixels wide by HDY pixels high. For example, the two dimensional arrayof pixels for the high definition spatial images 510 may be 1920 pixelswide by 1080 pixels high. The display devices 402L,402R in the stereoviewer 312 may display lower native resolution images 511N with atwo-dimensional array of pixels having a native resolution of LDX pixelswide by LDY pixels high. The dimensions LDX pixels wide and LDY pixelshigh of the lower native resolution images 511N are respectively lessthan the available spatial resolution of HDX pixels wide and HDY pixelshigh for the high definition spatial images 510.

The fovea 650 may be an image having dimensions FX pixels wide (X-axispixels) and FY pixels high (Y-axis pixels) of the high definition imagewithout interpolation or decimation such that there is no loss ofresolution or detail in the display area of interest to a surgeon. Inthis case there is a one to one mapping between pixels of the highdefinition image and pixels of the lower resolution display. However,extra pixels to each side of the fovea 650 need to be compressed ordecimated down to fewer pixels in the display.

For example, the high definition spatial images 510 are 1920 pixels wide(X-axis pixels) by 1080 pixels high (Y-axis pixels) and the native pixeldimensions of the display (low definition spatial images 511N) are 1280pixels wide (X-axis pixels) by 1024 pixels high (Y-axis pixels).Consider in this case that the fovea 650 is an image having dimensionsof 640 pixels wide (FX) and 512 pixels high (FY) (Y-axis pixels) to beplaced in the center of the display. An array of 640 pixels wide (X-axispixels) and 512 pixels high (Y-axis pixels) in the high definition image510 is mapped one to one into the 640 pixels wide (FX) (X-axis pixels)and 512 pixels high (FY) (Y-axis pixels) in the fovea 650. This leaves640 pixels wide (X-axis pixels) in the high definition image 510 to eachside of the fovea to be respectively mapped into 320 pixels wide (X-axispixels) to each side of the fovea in the display image 511N resulting ina two-to-one decimation if the full spatial image is to be displayed.Thus, a two-to-one decimation or compression in resolution maps theremaining X-axis pixels of the high definition image into the remainingX-axis pixels of the background or surround 651. Continuing with theY-axis pixels, 284 pixels high (Y-axis pixels) in the high definitionimage 510 above and below the fovea are to be respectively mapped into256 pixels high (Y-axis pixels) above and below the fovea in the displayimage 511N if the full spatial image is to be displayed. Thus,approximately a 1.1-to-1 decimation or compression in resolution alongthe Y-axis maps the remaining Y-axis pixels of the high definition imageinto the remaining Y-axis pixels of the background or surround 651. Notethat this assumes a total linear mapping in the surround 651, not apiece-wise linear in each of N quads, which may not work well in thecorners.

Note that with the total linear mapping in the surround 651 describedwith reference to FIG. 6C, the Y-axis compression or decimation maydiffer from the X-axis compression or decimation. In this case, theimage in the surround will be distorted by being compressed differentlyalong the axis with the greater decimation. In the case of the mappingsillustrated by FIGS. 6D-6E, the source/target windows are defined as apercentage of the source/target extent. Thus, the raw number of pixelsin the surround 651 differs in X,Y, but the percentage change betweenthe inner/outer windows is the same resulting in less distortion.

If the display is a high definition display with the same resolution ofhigh definition special images of the endoscopic camera, the background651 may be displayed at the native resolution while the fovea 650 isinterpolated up to be a magnified image within its pixel array of FX byFY pixels.

Automatic Digital and Mechanical Image Panning

In one embodiment of the invention, the fovea 650 may be fixed in thecenter of the display image 511N and the center of the display device.If the outer-source-window is smaller than the source extent, theinner/outer source windows may be digitally panned within the sourceframe. In this manner, inner/outer source window and the inner/outertarget windows are concentric to minimize distortion in thebackground/surround 651 around the fovea 650.

Alternatively in another configuration, the fovea 650 may be digitally(or electronically) moved within the display image 511N by various meansin response to an automatically sensed signal or a manually generatedsignal. That is, the fovea 650 may be digitally (electronically) pannedaround within the display image. This may be accomplished by changingthe coordinates defining the fovea 650 in the mapping of source pixelsto target pixels in the display. In this case, the inner/outer sourcewindow and the inner/outer target windows may not be concentric.

In either case, if an image is digitally panned without any mechanicalpanning of the endoscopic camera, the surgeon's perspective (angle atwhich the surgical site is viewed) on the surgical site is unchanged.

In the case of the moving fovea, if the fovea 650 nears the edge of thedisplay image 511N, a centralization process may occur where the pixelsof the display image 511N may adjust to position the fovea 650 morecentrally in the display image 511N. Moreover if the desired location offovea 650 is outside the matrix of pixels in the display image 511N, thedisplay image 511N may digitally adjust its position within the highdefinition spatial image 510 by selecting different pixels within thehigh definition spatial image 510. This is analogous to a virtual cameramoving around in the high definition spatial image 510. In this case,both the fovea 650 and the display image may be digitally(electronically) panned around within the matrix of pixels of the highdefinition spatial image 510.

In the alternate embodiment of the invention where the fovea 650 isfixed in the center of the display, the source window for selecting thesource of pixel information in the high definition video source imagesmoves to recenter the source area of interest within the fovea and thecenter of the display in a substantially instantaneous manner.

Further more, if the desired location of fovea 650 not only exceeds thepixels in the display image 511N but also the pixels of the highdefinition spatial image 510, the endoscopic camera 101B may bemechanically moved by the motors in the robotic arm 158B to adjust thefield of view of the surgical site in response thereto. In this case,the fovea 650 and the display image may be digitally (electronically)panned while the endoscopic camera 101B is mechanically panned to changethe field of view of the surgical site. In alternate embodiment of theinvention, the endoscopic camera 101B may be slewed slowly bothdigitally (electronically) and mechanically (physically) to maintain thesource area of interest substantially centered in the source videoframe. If the source area-of-interest is moved off-center, theendoscopic camera 101B may be mechanically moved and concurrently thesource window may be digitally moved in the opposite direction until thesource-window is re-centered relative to the full-extent of the sourcevideo captured by the endoscopic camera.

Reference is now made to FIGS. 7A-7D to illustrate digital panning ofimages and both digital and mechanical panning.

In FIG. 7A, an initial fovea position 650A of the fovea 650 is showncentered in an image 702A on a display 402L,402R. The pixels of image702A displayed by the display may be centered with respect to the pixelsof a high definition spatial image 700A providing the endoscopic camera101B field of view.

A surgeon or an assistant may desire to move the fovea 650 from theinitial fovea position 650A to a different fovea position 650B withinthe display image 511N or outside the display image 511N but within thehigh definition spatial image 700A. As mention previously, acentralization process may occur to select different pixels in thedisplay image 511N from the high definition spatial image to positionthe fovea 650 more centrally in the display image 511N, such asillustrated by the image 702B in FIG. 7B which has a different matrix ofpixels to display on the display 402L,402R. Within the display image511N and/or within the high definition spatial image 700A, the fovea 650is digitally moved from a first fovea position 650A displaying a firstarea of the surgical site to a second fovea position 650B displaying asecond area of the surgical site.

In FIG. 7B, the fovea position 650B is once again centered within theimage 702B that is displayed on the display 402L,402R. However, asurgeon or an assistant may desire to move the fovea 650 from thecentered fovea position 650B in FIG. 7B to a different fovea position650C outside of the display image 511N and the field of view of thesurgical site captured by the high definition spatial image 700Acorresponding to a given position of the endoscopic camera 101B. In thiscase, the endoscopic camera 101B may be mechanically panned to adifferent position to capture a different high definition spatial imageto display pixels of the desired fovea position 650C.

The camera control system of the robotic surgical system may first movethe fovea digitally. If the user out-paces the compensation rate ofre-centering the fovea digitally, the camera control systemtransitions/ramps to full endoscopic camera drive for the motors of therobotic surgical arm 101B to mechanically move the endoscopic camera.This may happen as the as the user out-paces the compensation rate ofthe slow re-centering loop that is attempting to keep the zoomedregion-of-interest centered in the video frame.

Note that moving an inner source window relative to an outer sourcewindow changes which pixels are mapped to the inner target window. Ifthe source frame region between the inner and outer source windows isbeing mapped to a surround on the target display, then moving the innersource window may also change the warp of the pixels that are mapped tothe surround. For example, in the surround the number of pixels mayexpand on one side while contracting on the opposite side.

As mentioned previously, the fovea 650 may be digitally moved from thefirst fovea position 650A to the second fovea position 650B within thedisplay image 511N and/or within the high definition spatial image 700A.The fovea 650 may be digitally moved abruptly from the first foveaposition 650A in one video frame to the second fovea position 650B inthe next video frame. Alternatively, the fovea 650 may be digitallymoved gradually from the first fovea position 650A to the second foveaposition 650B over a sequence of video frames with intermediate foveapositions there-between.

Referring now to FIG. 8, the first fovea position 650A and the secondfovea position 650B are illustrated with a plurality of intermediatefovea positions 850A-850D there-between. In this manner, the fovea 650may appear to move more gradually from the first fovea position 650A tothe second fovea position 650B within the display image 511N and/orwithin the high definition spatial image 700A.

Referring now to FIG. 7C, not only may the display image 511N bedigitally panned but the endoscopic camera 101B be mechanically panned.Additionally, a centering process that further adjust the digitalpanning of pixels and/or the mechanical panning of the endoscopic camera101B may be used to adjust the display image 511N to an image position702C around the fovea in order to center the desired fovea position 650Ctherein. In some cases, the centering process may be undesirable.

In FIG. 7D, the endoscopic camera 101B may be mechanically panned andthe display image 511N may be digitally panned to a image position 702Dbut without any centering process so that the desired fovea position650C is off-center within the display 402L,402R.

FIGS. 7C-7D illustrate combining digital image panning (digitaltracking) with mechanical camera panning (servo-mechanical tracking).The digital image panning (digital tracking) can be combined with themechanical camera panning (servo-mechanical tracking) analogous to amicro/macro mechanism or system. The digital image panning (digitaltracking) makes the relatively small and faster deviations or trackingefforts—digital in this case. The mechanical camera panning(servo-mechanical tracking) can handle larger deviations that occur moreslowly. Note that the effect of servo mechanical motion of the roboticsurgical arm 101B and the endoscopic camera 101B may be compensated. Thezoomed image or fovea 650 may be moved in the opposite direction of themovement of the endoscopic camera across the full special highdefinition image. In this case, the motion of the endoscopic camera 101Bmay be largely imperceptible when viewed in the zoomed image or fovea650.

While automatic panning of the endoscopic camera 101B is possible, itmay be preferable to avoid it and use digital panning alone. Otherwise,the endoscopic camera 101B may bump into something it should not unlessprecautions in its movement are taken. In this case, it is moredesirable to digitally pan the fovea 650 from one position to anotherwithout requiring movement of the endoscopic camera.

Automatic Camera Following and Manual Selection of Image Position

In some embodiments of the invention, it may be desirable to have theimage of the fovea or digital zoom area 650 automatically track orfollow some direct or indirect motions of the surgeon without moving theendoscopic camera 101B. In other embodiments of the invention, it may bedesirable to select the position of the fovea or digital zoom area 650within the background image 651 of the display. In still otherembodiments of the invention, it may be desirable combinecharacteristics of an automatic tracking system with a manual selectionsystem such as by setting preferences or making a choice regarding thefovea or digital zoom area 650 and allow it to track a surgeon's motionin response thereto.

Automatic camera following and digital zoom are combined together suchthat the digital zoomed portion of an image tracks or follow a surgeon'smotions, such as the gaze of his pupils, without requiring mechanicalmovement of the endoscopic camera. If the surgeon's motions indicatethat the digital zoomed portion extend beyond pixels of the highdefinition digital image being captured, the endoscopic camera may bemechanically moved automatically.

For automatic camera following, different sensing modalities may be usedto detect a surgeon's motion so that a digital zoomed portion ofinterest of an image may be moved around within the pixels of a highdefinition digital image. Some different sensing modalities include (1)robotic surgical tool tracking, (2) surgeon gaze tracking; (3) or adiscrete user interface.

Robotic surgical tool tracking may be performed by kinematics sensingthrough joint encoders, potentiometers, and the like; videoanalysis-based tool location tracking; or a combination or fusion ofkinematics sensing and video analysis-based tool location tracking.Robotic surgical tool tracking is further disclosed in U.S. patentapplication Ser. No. 11/130,471 entitled METHODS AND SYSTEM FORPERFORMING 3-D TOOL TRACKING BY FUSION OF SENSOR AND/OR CAMERA DERIVEDDATA DURING MINIMALLY INVASIVE ROBOTIC SURGERY filed by Brian DavidHoffman et al. one May 16, 2005, which is incorporated herein byreference and in U.S. patent application Ser. No. 11/865,014 entitledMETHODS AND SYSTEMS FOR ROBOTIC INSTRUMENT TOOL TRACKING filed by WenyiZhao et al. on Sep. 30, 2007, which is also incorporated herein byreference.

Referring now to FIGS. 17A-17B, a centroid (tool centroid) 1701 for therobotic surgical tools 510L,510R may be determined from the respectiveposition information points 1710L,1710R within the surgical sitedetermined from a tool tracking system. The tool centroid 1701 may beused as a center point to automatically position the center of the fovea650 (re-center) within the image 511N.

For example, the robotic surgical tool 510R may shift in the surgicalsite to a position indicated by the robotic surgical tool 510R′. Theposition information follows the change in position of the tool to therespective position information point 1710R′. A new position of toolcentroid 1701′ is determined given the position information points1710L,1710R′. This makes the fovea 650 off-center from the new positionof the tool centroid 1701′. The new position of the tool centroid 1701′may be used as a center point to automatically re-center the fovea 650within the image 511N.

FIG. 17B illustrates the fovea 650 re-centered within the image 511N inresponse to the new position of the tool centroid 1701′.

A discrete user interface may be provided to a surgeon at the mastercontrol console to control the position of the fovea 650 within theimage 511N of the display. One or more buttons (such as arrow buttons tothe side of a surgeon's console), one or more foot pedals, or the mastercontrol handles 160 themselves may be used to manipulate the position ofthe fovea 650 or other image. A voice recognition system at the mastercontrol console capable of recognizing vocal commands may also be usedto adjust the position of the fovea 650.

One or more buttons, foot pedals, or combinations thereof may be pressedto manually move the fovea 650 or other images up, down, left, and/orright. Voice commands may be used in another configuration to move thefovea 650 or other images up, down, left, and/or right.

Alternatively, the discrete user interface may be used to actuate anautomatic re-centering process of the digital zoomed image 650 based oncurrent tool position, gaze location, or other available information inthe surgical system. Alternatively, the discrete user interface may beused to re-center or move the image at discrete times, such as throughvoice activation, perhaps in concert with tool tracking or gazedetection.

As mentioned herein, the master control handles 160 themselves may beused to manipulate the position of the fovea 650 or other image. In sucha case, one or both, of the master control handles 160 can serve as atwo-dimensional or three-dimensional mouse (masters-as-mice).Accordingly, one or both of the master control handles 160 can bearranged to perform functions relative to the fovea image 650 in amanner analogous to a conventional mouse relative to a computer screen.

Each of the master control handles 160 may have at least six degrees offreedom of movement. Accordingly, when used as a three-dimensionalmouse, a master control handle can be arranged to control six variables,for example. Therefore, functions such as, shifting, rotating, panning,tilting, scaling, and/or the like, can be performed simultaneously whenone, or both, or either, of the masters are used as a three-dimensionalmouse, without another input being required. In particular, fortwo-handed or two-master operation, any windows or overlays can behandled as “elastic” bodies, such that resizing, scaling, warping,and/or the like, can, for example, be controlled by pulling the mastersapart, or the like.

One or both of the master control handles 160 may select and drag thefovea to different positions within the image 511N, either by adjustingits size/position within the image 511N, and/or by defining a croprectangle to generate the fovea 650 from the background image 651representative of the full spatial high definition images. Themasters-as-mice functionality of the master control handles 160 cansupport successive refinement of the position of the fovea as well ascontrol the level of image magnification or zoom within the highdefinition images.

In yet another configuration, the robotic surgical tools may be used todrag the fovea 650 to different positions within the image 511N and/ormove the image 511N within the matrix of pixel information of the highdefinition images.

Referring now to FIG. 18A, robotic surgical tool 510R has a positioninformation point 1810 well away from the edge and closer to center ofthe fovea 650. A tool tracking system may be used to provide theinformation regarding the position information point 1810R of therobotic surgical tool relative to the endoscopic camera 101B. A surgeonmay desire to move the fovea 650 within the image 511N to better magnifya different location within the surgical site. In this case, the roboticsurgical tool 510 may act as a poker to poke or bump an edge of thefovea 650 to move up, down, left, right, and/or combinations thereofwithin the image 511N.

In an alternate embodiment of the invention with the fovea 650 in afixed position in the center of the display, an elastic wall or otherhaptic interface may be simulated such that when the robotic surgicaltool bumps into the outer edge of the fovea, or outer edge of the targetwindow, the center position of the source area-of-interest pansaccordingly to be within the fovea 650.

In FIG. 18A, the robotic surgical tool 510R has moved in position torobotic surgical tool position 510R′ with the position information point1810R′ near the edge of the fovea 650. The digital zoom/panning systemmay pan the fovea 650 in response to the robot surgical tool being inthe robotic surgical tool position 510R′ with the position informationpoint 1810R′ substantially near the edge of the fovea 650.

Referring now to FIG. 18B, the fovea 650 has panned from its position inFIG. 18A to the fovea position 650′ so that the robotic surgical toolposition 510R′ and position information point 1810R′ are more centeredwithin the fovea. However, a surgeon may desire to move from the foveaposition 650′ to another position. In this case, the surgeon may use therobotic surgical tool again to pan the fovea 650. The robotic surgicaltool 510R has moved in position from the robotic surgical tool position510R′ to the robotic surgical tool position 510R″ with the positioninformation point 1810R″ near the top edge of the fovea 650. In thiscase, the fovea 650 will be panned up from its position 650″ in FIG. 18Bso that the robotic surgical tool position 510R″ and positioninformation point 1810R″ will be more centered within the fovea.

One or more of the manual user interface techniques may be combined withan automatic user interface technique for digital panning/zooming.

Gaze Detection and Tracking

One of the sensing modalities that may be used for automatic camerafollowing or image panning is gaze tracking of a surgeon's eyes in thestereo viewer 312.

As described with reference to FIGS. 4A-4C, the stereo viewer 312 mayinclude one or more left gaze detection sensors 420L near the peripheryof the display device 402L for the left eye of the surgeon and one ormore right gaze detection sensors 420R near the periphery of the displaydevice 402R for the right eye of the surgeon. One of the gaze detectionsensors for each eye may also include a low level light source 422L,422Rto shine light into the eye of the surgeon to detect eye movement withthe respective gaze detection sensors 420L,420R.

The one or more left gaze detection sensors 420L and the one or moreright gaze detection sensors 420R are used to determine the location ofthe central gaze of the surgeon's eyes within the image that isdisplayed on the display devices 402L,402R respectively. The centralgaze location within the image may be used to define the center point ofthe fovea 650 within the image 511N. As the surgeon's gaze moves aroundwith the image 511N, the fovea 650 may digitally move as well to providea magnified image where the surgeon is gazing. Moreover, if the surgeongazes in a location for a predetermined period of time, that area of theimage may be digitally and/or mechanically automatically re-centeredwithin the image 511N on the display devices 402L,402R. If instead thefovea 650 is in a fixed position in the center of the display, thesurgeon's gaze off center of the image 511N for a predetermined periodof time may shift the source area of interest to be in the center of thedisplay within the fovea 650.

Exemplary algorithms for gaze detection and tracking are described indetail in “Gaze Contingent Control for Minimally Invasive RoboticSurgery” by Mylonas G. P., Darzi A, Yang G-Z. Computer Aided Surgery,September 2006; 11(5): 256-266; “Visual Search: Psychophysical Modelsand Practical Applications” by Yang G-Z, Dempere-Marco L, Hu X-P, RoweA. Image and Vision Computing 2002; 20:291-305; and “Gaze ContingentDepth Recovery and Motion Stabilisation for Minimally Invasive RoboticSurgery” by George P. Mylonas, Ara Darzi, Guang-Zhong Yang; MIAR 2004,LNCS 3150, pp. 311-319, 2004. Exemplary algorithms for gaze detectionand tracking are also described in U.S. Pat. No. 5,912,721 which isincorporated herein by reference.

The digitally formed fovea 650 and the digital panning of the foveawithin the image 511N in response to gaze detection, allows theendoscopic camera 101B to remain stationary, at least for smalladjustments. The automatic digital panning of the fovea 650 with thefull spatial high definition image of the endoscopic camera in thebackground 651, a surgeon is less likely to be interrupted duringsurgery to change the view of images. That is, with the automaticdigital panning of the fovea 650 and the full spatial high definitionimage in the background 651, a surgeon may avoid having to change theview of the surgical site by manual manipulation of the robotic arm 101Band the endoscopic camera. A decrease in surgeon interruption to changethe view and manipulate the camera can improve the efficiency of therobotic surgical system.

Referring now to FIG. 9, a face is illustrated with stereo gazedetection about the left and right eyes to detect left and right pupilpositions for gaze detection. The sensors may sense the pupil positionswith respect to the left, right, top, and bottom edges of the eye. InFIG. 9, a surgeon may initially gaze directly ahead at a test pattern tocalibrate the gaze detection system with left and right eyes gazing to acenter position.

In contrast with the center position of FIG. 9, FIG. 11A illustratesleft and right eyes gazing to an upper left position. FIG. 11Billustrates left and right eyes gazing to a lower right position.

The gaze of the pupils can be detected in a number of different ways.FIG. 10 illustrates exemplary left and rights graphs 1002L,1002R as tohow the edges of the pupil may be sensed with respect to the top,bottom, left, and right corners 1001T, 1001B, 1001L, 1001R of the leftand right eyes 1000R, 1000L.

The edge images for the right eye and left eye of may be formed viaknown methods, such as a Sobel filter or a Canny filter. The edge imagescan then be mapped in a direction perpendicular to the one-dimensional(1D) axis direction to detect the inner corners of the eyes. The imagecan then be scanned in a direction normal to the 1D-axis, with thelowest brightness point being the point of the inner corner of the eye.The peaks in the brightness points on the graphs 1002L,1002R mayindicate the position of the edges of the left and right pupils.

As the pupils move horizontally left or right, the position of the peaksalong the graphs 1002R, 1002L shift respectively left or right. Similargraphs may be generated for vertical movement of the pupils up and down.

It may be desirable to detect head movement within the stereo viewer 312for a more accurate gaze detection system. Head movement may be detectedby one or more head motion sensors or algorithmically by using one ormore gaze detection sensors 420L,420R. The level of head motion detectedmay be removed from gaze detection signals so that inadvertent headmovement does not result in movement of the fovea 650 within the image511N.

Referring now to FIG. 12, vertical head movement illustrated by arrow Amay be detected by monitoring the movement of a line 1200 formed throughthe corners 1001L, 1001R of the left and right eyes. The corners of theleft and right eyes may be determined from the edge images of the eyes.

Referring now to FIG. 13, a combination of vertical and horizontal headmovement may be detected using at least two corners 1001T, 1001B, 1001L,1001R of the left and right eyes. The top corner 1001T and the leftcorner 1000L of the right eye 1000R and the top corner 1001T and theright corner 1000R of the left eye 1000L may be used to form a polygonhaving a centroid. The centroid moves along a vector. The corners of theeyes may be monitored to detect movement in the centroid and the vectorso that a combination of vertical and horizontal head movement may bedetected.

Automatic Zoom Level

A surgeon may desire additional zoom or magnification of an objectdisplayed in the fovea 650. Alternatively, the surgeon may desire lesszoom or demagnification of an object displayed in the fovea 650. Thelevel of the level of zoom may be set by manually by the selection ofrelative sizes of the source windows 661 and target windows 671illustrated in FIG. 6D. However, methods of automatically determining anappropriate level of zoom may be made by automatically determining therelative sizes of the source windows 661 and target windows 671.

An approximation for the desired depth of the fovea 650 may beautomatically determined by an average extent of instrument motion. Theaverage extent may be determined by making a time weighted average ofthe motion in the robotic surgical instruments. Such extent defines abox or area within the image 511N or display 402L,402R. A determinationof the minimum zoom that can display the box or area defined by theextent may be the appropriate level of zoom to select.

Gaze detection may also be used to automatically determine anapproximation for the desired depth of the fovea 650. As the surgeonseyes move over the background 651 in the image 511N, the gaze motion ofthe surgeon's pupils or eyes may be stored over time. A time-weightedaverage of the stored gaze motion can be computed to automaticallydefine a two dimensional area or a three dimensional surface within theimage 511N or display 402L,402R. A determination of the minimum zoomthat can display the two dimensional area or the three dimensionalsurface defined by the extent of the gaze motion of the surgeon's eyesmay be the appropriate level of zoom to select.

In another configuration, the boundary defined by illumination falloffmay be used to automatically select the source area of interest fordisplay within the fovea 650.

If an automated digital panning occurs of the fovea 650 or the imageunder the fovea 650, the digital zoom may momentarily zoom out from thearea of interest and then zoom back when the area of interest issubstantially centered in the fovea 650.

A macro/micro approach can also be adapted along the insertion axis 574(see FIG. 1C) of the endoscopic camera 101B mounted on the roboticsurgical arm 158B. The endoscopic camera 101B may be physically andmechanically moved in and out of the surgical site along the insertionaxis 574 by the motor 574 providing a macro adjustment. Howeverinitially from a fixed position, if the surgeon wishes to see a slightlynarrower field of view, the camera can be virtually moved in along theinsertion axis toward the tissue by increasing the digital zoom factorproviding a micro adjustment, by decreasing the size of thearea-of-interest selected from the source high definition video images.In this case, the endoscopic camera is virtually (electronically) movedby digital signal processing of the source video images without anyphysical or mechanical movement.

When the digital zoom exceeds a predetermined limit or the source windowcrosses over a predetermined lower size limit, the motor 574 may beengaged to physically and mechanically moved the endoscopic camera 101Balong the insertion axis 574 to avoid an interpolation or a level ofinterpolation of the pixels (source pixels) in the source highdefinition video. This is analogous to mechanically moving (clutching)the camera along yaw/pitch axes when the fovea reaches the edge of thehigh definition video source. Alternately, endoscopic camera could beslowly adjusted along the insertion axis both electronically digitallyand physically so as to maintain a source area-of-interest at apercentage (e.g., approximately 50%) of the source frame size. This isanalogous to a slow slew/auto-recentering of the fovea.

The zoom factor for the fovea 650 may also be automatically determinedby a distance from the end of the endoscopic camera to the operativesite within the surgical cavity. This is analogous to auto-focus methodsin digital cameras and how they derive an estimate of the working depthof focus.

Display Panel User Interface

Much of the discussion regarding digital zooming and digital panning iswith regards to a surgeon O at the controls 160 of the master console150. The same images seen by the surgeon in the stereo viewer may bemonitored by an assistant on the external monitor 154 illustrated inFIGS. 1A-1B. However, the assistant A may also choose to see a differentimage than that of the surgeon without moving the endoscopic camera. Theassistant A can control a second digital zoom and a second digital panof the captured high definition digital images from the endoscopiccamera 101B so that they can display a different view of images of thesurgical site on a second display device, the external monitor 154. Theassistant A may control the selection of the second digital zoom and thesecond digital pan on the monitor 154 in a number of ways.

Referring now to FIG. 14, the external monitor 154 may include a touchscreen or touch panel interface 1401 to control the selection of thesecond digital zoom and the second digital pan on the monitor 154. Forexample, the assistant may touch his finger to the touch panel 1401 andselect a region of the display to be the target window or fovea 650 witha linear digital zoom. With the fovea 650 defined and in a fixedposition on the display, the assistant may then use one or more fingersF to scroll the image under the fovea to display a desired region ofinterest in the surgical site captured by the high definition sourcevideo images. Alternatively, a predetermined rectangular shape may bemoved over the image on the touch panel with a finger F to select thedesired region of interest to position within a fovea in the center ofthe display monitor 154. With the finger F on the touch panel 1401, thefull frame image may be momentarily displayed on the touch panel 1401 sothat the region of interest may be selected and then pop back out tozoomed-in view with the desired magnification of the fovea. In thesecases, the assistant does not need to mechanically move the endoscopiccamera 101B, avoiding clutching the robotic surgical arm 158B tophysically move the endoscopic camera to another position.

Alternatively, one or more control buttons 1404A-1404B may be providedby the monitor 154 to digitally zoom and magnify the image provided bythe fovea 650 or to digitally move the center of the fovea to anotherposition within the surgical site. Up, down, left, and right pan arrows1406 may be provided to pan the fovea within the captured pixels of theendoscopic camera to display a different fovea 650 within the image511N.

In another configuration, the assistant may control the digital pan andthe digital zoom for the fovea within the image by physical movement ofthe monitor 154. In this case, the monitor may include an inertia sensor1450 to detect movement from an initial position 154A to variousdifferent positions such as positions 154B-154C illustrated in FIG. 15.For example, the inertia sensor 1450 may detect movement in the X andY-axes to pan the fovea 650 around the image 511N displayed on themonitor 154. The inertia sensor 1450 may detect movement in the Z axisto zoom the fovea 650 in and out of the image 511N displayed on themonitor 154, for example.

Referring now to FIG. 15, a support arm 1501 includes a plurality oflinks 1505 to moveably support the monitor 154 coupled to the side cart152. At a plurality of joints 1512 between the links 1505, the supportarm includes a plurality of encoders 1510 in accordance with anotherembodiment of the invention.

In this case, the position of the monitor 154 is determined by theencoders 1510. The assistant may physically move the monitor 154 bygrabbing it with their hands H1-H2. The movement in the monitor istranslated to the joints through the links of the support arm 1501 andsensed by the encoders 1510. The encoders 1510 can detect movement froman initial position 154A to various different positions of the monitor154 such as positions 154B-154C in order to digitally pan or digitallyzoom the fovea 650. In this manner, intuitive camera control can beprovided to the assistant, as an alternative to mechanically moving thecamera with the camera clutch.

As another aspect of the invention, the monitor 154 may also be movedalong and rotated about the axes to possibly control the movements of arobotic surgical tool 101, such as during initial set up or duringsurgery to control an extra tool, such as a suction tool for example.Another extra robotic surgical tool that may be controlled by anassistant is an ultrasound tool. The images generated by the ultrasoundtool can be displayed on the monitor 154 as well the display devices402L,402R in the stereo viewer 312. As the ultrasound tool is moved oversurfaces in the surgical site, the ultrasound images that are displayedchange.

System and Operational Methods

Referring now to FIG. 16, a functional block diagram of a digital videozoom subsystem 1600 is illustrated. The subsystem 1600 is an aspect ofthe robotic surgical system that may provide the digital zoom portion ofvideo information and the automatic panning of video information in asurgical site.

The subsystem 1600 may include an image acquisition device (endoscopiccamera) 1602, an image buffer 1604, a first digital mapper and imagefilter 1606A, a first user interface 1608A, a first display buffer1610A, and a first display device 1612A coupled together as shown. Thefirst display device 1612A may be one of the display device 154 or thestereo display devices 402L,402R, for example. The subsystem 1600 mayfurther include a second digital mapper and image filter 1606B, a seconduser interface 1608B, a second display buffer 1610B, and a seconddisplay device 1612B coupled together as shown and independent of thefirst devices.

The image acquisition device 1602 may capture images of a surgical sitein a high definition image format. The image buffer 1604 buffers one ormore frames of a matrix of pixel data. The first digital mapper andimage filter 1606 may map and filter the pixels in the captured imagesto properly display pixels on the first display device 1612A as desired.The first display buffer 1610 is coupled between the image filter 1606and the first display device 1612A to store one or more frames of pixelinformation for display on the display device.

The first user interface 1608A may include a region of interest (fovea)selector 1620, a user preference selector 1622, and an enhanced displaymode selector 1624 to select an enhanced display mode 1634. The regionof interest (fovea) selector 1620 may function similar to the method andapparatus for automatic digital panning of the fovea 650 as describedpreviously. A user may select how the source rectangle shouldautomatically adjust its position with respect to an estimated toolcentroid 1630, depth 1631, user focal-point, or mean working envelope,for example. The user preference selector 1622 allows a user to manuallyselect the source data from a source rectangle 1632, such as afull-spatial high definition image, and manually select the destinationrectangle 1633 for where the image may be preferably displayed on thefirst display device 1612A. Without the enhanced display mode beingselected, the user may manually select the source rectangle 1632 and thedestination rectangle 1633. If the system is selected to be in anenhanced display mode, the source rectangle 1632 and/or the destinationrectangle 1633 may be automatically selected based on one or more of theestimated tool centroid 1630, the depth 1631, the user focal-point, orthe mean working envelope. In some cases, a user may select a fixeddestination rectangle while the source rectangle 1632 is automaticallyselected.

As the image acquisition device 1602 captures digital pixel data ofimages of a surgical site that are stored in the image buffer 1604, thepixel data can be independently selected for viewing by multiple displaydevices.

The second digital mapper and image filter 1606B, the second userinterface 1608B, and the second display buffer 1610B are for independentselection and display of images on the second display device 1612B. Forexample, the first display 1612A may be the stereo display devices402L,402R in the console 150 while the second display 1612B may be theassistant's display device 154 illustrated in FIG. 1A. A first user mayindependently select user preferences for the first display with thefirst user interface 1608A, while a second user may independently selectuser preferences for the second display with the second user interface1608B. The second user interface 1608B is substantially similar to thefirst user interface 1608A and its description is incorporated herein byreference for brevity. Alternatively, the second digital mapper andimage filter 1606B, the second user interface 1608B, and the seconddisplay buffer 1610B may be synchronized to the first devices such thatthe display of images on the second display device 1612B are similar tothe display of images on the first display device 1612A.

CONCLUSION

The embodiments of the invention have now been described.

A number of elements of the system may be implemented in software andexecuted by a computer and its processor, such as computer 151 and itsprocessor 302. When implemented in software, the elements of theembodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable medium or transmitted by a computer data signalembodied in a carrier wave over a transmission medium or communicationlink. The processor readable medium may include any medium that canstore or transfer information. Examples of the processor readable mediuminclude an electronic circuit, a semiconductor memory device, a readonly memory (ROM), a flash memory, an erasable programmable read onlymemory (EPROM), a floppy diskette, a CD-ROM, an optical disk, a harddisk, a fiber optic medium, a radio frequency (RF) link, etc. Thecomputer data signal may include any signal that can propagate over atransmission medium such as electronic network channels, optical fibers,air, electromagnetic, RF links, etc. The code segments may be downloadedvia computer networks such as the Internet, Intranet, etc.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat the embodiments of the invention not be limited to the specificconstructions and arrangements shown and described, since various othermodifications may become apparent after reading the disclosure. Forexample, while the inner/outer pair of source windows 661 andinner/outer pair of target windows 671 have been shown and described asbeing rectangular in shape, they may be circular in shape in alternateembodiments of the invention. Additionally, some embodiments of theinvention have been described with reference to a video system in arobotic surgical system. However, these embodiments may be equallyapplicable to other video systems. Thus, the embodiments of theinvention should be construed according to the claims that follow below.

What is claimed is:
 1. A digital zoom and panning system, the systemcomprising: an endoscopic camera device to capture digital video imagesof a surgical site; a left display device having a left set of displaypixels to display images; a right display device, adjacent the leftdisplay device, the right display device having a right set of displaypixels to display images; a first user interface displayed on the leftdisplay device and the right display device to accept a user input todisplay a fovea, the user input including selection of a source pixelarray of source pixels within a first frame of the digital video imageswith reference to the surgical site and selection of a target pixelarray of target pixels within a subset of the left and right set ofdisplay pixels of the left and right display devices; a left digitalmapping and filtering device coupled between the endoscopic camera andthe left display device, the left digital mapping and filtering deviceto selectively map and filter source pixels in a first region ofinterest of the digital video images into target pixels in a leftdestination rectangle for the left display device in response to theuser selection; and a right digital mapping and filtering device coupledbetween the endoscopic camera and the right display device, the rightdigital mapping and filtering device to selectively map and filtersource pixels in the first region of interest into target pixels in aright destination rectangle for the right display device in response tothe user selection; wherein the images displayed on the first and seconddisplay devices are displayed in stereo.
 2. The digital zoom and panningsystem of claim 1, further comprising: a left image forming device and aright image forming device coupled to the endoscopic camera deviceconfigured to generate a left video channel and a right video channel offrames of images of the surgical site, respectively.
 3. The digital zoomand panning system of claim 2, further comprising: A left camera controlunit and a right camera control unit coupled between the left and rightimage forming devices and the left and right display devices, a lockreference signal coupled between the left and right camera control unitsto synchronize the left video channel to the right video channel.
 4. Thedigital zoom and panning system of claim 1, wherein: the first digitalmapping and filtering device to automatically select a first region ofinterest within the digital video images to display in the firstdestination rectangle; and the first digital mapping and filteringdevice selectively maps and filters the source pixels in the firstregion of interest into target pixels of the first display device. 5.The digital zoom and panning system of claim 4, wherein the automaticselection of the first region of interest is in response to a toolcentroid, a depth of focus, a user focal-point, an average workingenvelope, or a combination thereof.
 6. The digital zoom and panningsystem of claim 1, further comprising: an image buffer coupled to theendoscopic camera device, the image buffer to store one or more framesof the digital video images as source pixels.
 7. The digital zoom andpanning system of claim 1, further comprising: a left display buffercoupled to and between the left display device and the left digitalmapping and filtering device, the left display buffer to store one ormore frames of target pixels received from the left digital mapping andfiltering device and write one or more frames of target pixels to theleft display device to display images; and a right display buffercoupled to and between the right display device and the right digitalmapping and filtering device, the right display buffer to store one ormore frames of target pixels received from the right digital mapping andfiltering device and write one or more frames of target pixels to theright display device to display images.
 8. The digital zoom and panningsystem of claim 1, further comprising: a second user interface to acceptsecond user input including the second source rectangle to select secondsource pixels within frames of the digital video images, the seconddestination rectangle to select target pixels within the second displaydevice to display images, and the second region of interest within thedigital video images to display in the second destination rectangle. 9.The digital zoom and panning system of claim 1, wherein the imageacquisition device is a stereo endoscopic camera to capture digitalvideo images of a surgical site and to be mechanically panned within thesurgical site; and in response to the detection of the first region ofinterest being outside the first source rectangle, the robotic stereoendoscopic camera to mechanically pan within the surgical site and thefirst digital mapping and filtering device to digitally pan and displayimages in the first region of interest into the first destinationrectangle.
 10. A digital zoom and panning method for digital video, themethod comprising: capturing digital video images of a surgical sitewith an endoscopic camera device; storing one or more frames of thedigital video images as source pixels; displaying images on a displaydevice, the display device including a plurality of display pixels todisplay the images; displaying a user interface on the display device;receiving a user input to selectively display a navigation window on thedisplay device, the user input including a selection of a source pixelarray of source pixels within frames of the digital video images and aselection of a target pixel array of target pixels within the pluralityof display pixels of the display device; down-sampling the source pixelsin a source rectangle of the digital video images into the target pixelsin the target pixel array to form an image for the navigation window;and displaying the image for the navigation window within the pluralityof display pixels of the display device; wherein the navigation windowshows a pull-back view of the surgical site.
 11. The digital zoom andpanning method of claim 10, wherein the down-sampled image for thenavigation window is displayed picture-in-picture (PIP) within a smallerdisplay region of the display device.
 12. The digital zoom and panningmethod of claim 10, further comprising: automatically selecting a firstregion of interest within the digital video images to display in a firstdestination rectangle; and selectively mapping and filtering the sourcepixels in the first region of interest into target pixels of the firstdisplay device.
 13. The digital zoom and panning method of claim 12,wherein the automatic selecting of the first region of interest is inresponse to a tool centroid, a depth of focus, a user focal-point, anaverage working envelope, or a combination thereof.
 14. The digital zoomand panning method of claim 10, wherein: storing one or more frames ofdigital image, stores the frames to an image buffer coupled to andbetween the endoscopic camera device and a digital mapping filteringdevice.
 15. The digital zoom and panning method of claim 14, furthercomprising: displaying images on a second display device; andselectively mapping and filtering source pixels in a second region ofinterest from the image buffer to digitally zoom into target pixels in asecond destination rectangle of the second display device.
 16. A digitalzoom and panning method for digital video, the method comprising:capturing digital video images of a surgical site with an endoscopiccamera device; storing one or more frames of the digital video images assource pixels; displaying images stereoscopically on a left displaydevice and a right display device, the left and right display devicedisplaying a plurality of display pixels to display the images;displaying a user interface on the left and right display devices;receiving a user input to selectively display a navigation window on theleft and right display devices, the user input including a selection ofa source pixel array of source pixels within frames of the digital videoimages and a selection of a target pixel array of target pixels withinthe plurality of display pixels of the display device; down-sampling thesource pixels in a source rectangle of the digital video images into thetarget pixels in the target pixel array to form an image for thenavigation window; and displaying the image for the navigation windowwithin the plurality of display pixels of the display device; whereinthe navigation window shows a pull-back view of the surgical site. 17.The digital zoom and panning method of claim 16, wherein thedown-sampled image for the navigation window is displayedpicture-in-picture (PIP) within a smaller display region of the displaydevice.
 18. The digital zoom and panning method of claim 16, furthercomprising: automatically selecting a first region of interest withinthe digital video images to display in a first destination rectangle;and selectively mapping and filtering the source pixels in the firstregion of interest into target pixels of the first display device. 19.The digital zoom and panning method of claim 18, wherein the automaticselecting of the first region of interest is in response to a toolcentroid, a depth of focus, a user focal-point, an average workingenvelope, or a combination thereof.
 20. The digital zoom and panningmethod of claim 16, wherein: storing one or more frames of digitalimage, stores the frames to an image buffer coupled to and between theendoscopic camera device and a left digital mapping filtering device anda right digital mapping filtering device.